Method and apparatus for recognizing location of user

ABSTRACT

A method of recognizing a location of a user including detecting the user&#39;s two eyes and mouth of their face is provided, which includes calculating a ratio of a distance between the two eyes to a distance between a middle point of the two eyes and the mouth, calculating a rotation angle of the face according to the ratio, and detecting a distance between the face and the camera based on the rotation angle.

PRIORITY

This application claims priority under 35 U.S.C. §119(a) to KoreanPatent Application No. 10-2010-0043058, filed on May 7, 2010, in theKorean Intellectual Property Office, the disclosure of which isincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to a method and apparatus forrecognizing a location of a user, and more particularly, to a method andapparatus for recognizing a location of a user using a single camera.

2. Description of the Related Art

In order to acquire information about a current location of a user,camera hardware using a stereo method, a Time of Flight (TOF) method, orthe like is generally used. However, since such methods requirecalibration and matching, the camera hardware using these methods aregreatly affected by illumination. Also, when applied to consumerelectronics, the camera hardware required for these methods hinderstheir price competitiveness.

SUMMARY OF THE INVENTION

The present invention provides a method and apparatus for accuratelyrecognizing a location of a user using a single camera.

According to an aspect of the present invention, there is provided amethod of recognizing a location of a user using a camera, the methodincluding detecting two eyes and a mouth of the user's face; calculatinga ratio of a distance between the two eyes to a distance between amiddle point of the two eyes and the mouth; calculating a rotation angleof the face according to the ratio; and detecting a distance between theface and the camera based on the rotation angle.

The method may further include generating a zoom-in signal or a zoom-outsignal with respect to the face based on the distance between the faceand the camera.

The method may further include a zoom-in signal or a zoom-out signalwith respect to contents displayed on an image display device connectedto the camera based on the distance between the face and the camera.

The method may further include a signal for changing a location of animage display device connected to the camera based on the rotationangle.

The method may further include calculating an angle between a lineconnecting the two eyes and a reference line, wherein, when it isassumed that the user directly faces the camera, the reference linerefers to a line connecting two virtual eyes.

The method may further include generating a signal for rotating contentsdisplayed on an image display device connected to the camera based onthe angle between the line connecting the two lines and the referenceline.

The method may further include a signal for changing a location of animage display device connected to the camera based on the angle betweenthe line connecting the two eyes and the reference line.

According to another aspect of the present invention, there is providedan apparatus for recognizing a location of a user using a camera, theapparatus including a detecting unit that detects two eyes and a mouthof the user's face; and a calculating unit that calculates a ratio of adistance between the two eyes to a distance between a middle point ofthe two eyes and the mouth, calculates a rotation angle of the faceaccording to the ratio, and calculates a distance between the face andthe camera based on the rotation angle.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages of the present inventionwill become more apparent from the following detailed description takenin conjunction with the accompanying drawings, in which:

FIG. 1 is a block diagram of an apparatus for recognizing a location ofa user, according to an embodiment of the present invention;

FIG. 2 is a flowchart illustrating a method of recognizing a location ofa user, according to an embodiment of the present invention;

FIG. 3 is a view for explaining a ratio of a distance between two eyesto a distance between a middle point of the two eyes and mouth,according to an embodiment of the present invention;

FIGS. 4 and 5 illustrate rotation angles of a face according toembodiments of the present invention;

FIG. 6 illustrates a first lookup table according to an embodiment ofthe present invention;

FIG. 7 illustrates a second lookup table according to an embodiment ofthe present invention;

FIG. 8 illustrates a third lookup table according to an embodiment ofthe present invention;

FIG. 9 is a flowchart illustrates a method of recognizing a location ofa user, according to another embodiment of the present invention;

FIG. 10 illustrates an angle between a line connecting the two eyes anda reference line, according to an embodiment of the present invention;

FIGS. 11 and 12 illustrate rotations of contents and an image displaydevice according to an angle between a line connecting the two eyes anda reference line, according to embodiments of the present invention; and

FIG. 13 illustrates a rotation of an image display device according to arotation angle, according to an embodiment of the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

The present invention will now be described more fully with reference tothe accompanying drawings, in which embodiments of the invention areshown. Embodiments disclosed in the specification and structures shownin the drawings are merely exemplary of the present invention, and itshould be understood that variations capable of substituting for theembodiment may exist at the time of filing the application.

FIG. 1 is a block diagram of an apparatus 110 for recognizing a locationof a user, according to an embodiment of the present invention.

Referring to FIG. 1, the apparatus 110 includes a camera 111, adetecting unit 112, a calculating unit 113, a storage unit 114, and acontrol unit 115. In a hardware configuration, the detecting unit 112,the calculating unit 113, the storage unit 114, and the control unit 115may be included in the camera 111. Also, the apparatus 110 may beembedded in an image display device 120.

The camera 111 captures an image of the user who is a subject to beprocessed, and the detecting unit 112 detects the two eyes and mouth ofthe user.

The calculating unit 113 calculates a distance between the two eyes anda distance between a middle point of the two eyes and the mouth. Next,the calculating unit 113 calculates a ratio of the distance between thetwo eyes to the distance between the middle point of the two eyes andthe mouth, calculates a rotation angle of a face according to the ratio,and detects a distance between the face and the camera 111 based on therotation angle.

More specifically, the calculating unit 113 calculates a rotation angleof a face by using a first lookup table in which a predeterminedrotation angle according to a calculated ratio is stored, calculates adistance between two virtual eyes by using a second lookup table inwhich information obtained by converting the distance between the twoeyes into a distance between two virtual eyes when the user directlyfaces the camera 111 is stored, and detects a distance between the faceand the camera 111 by using a third lookup table in which a realseparation distance between the face and the camera 111 according to thedistance between the two virtual eyes is stored.

A distance between two virtual eyes refers to a distance between twoeyes when it is assumed that a user directly faces the camera 111. Also,the calculating unit 113 calculates an angle between a line connectingthe two eyes and a line connecting the two virtual eyes when it isassumed that the user directly faces the camera 111.

The storage unit 114 stores the first lookup table, the second lookuptable, and the third lookup table.

The control unit 115 generates a zoom-in signal or a zoom-out signalwith respect to the user's face based on the distance between the user'sface and the camera 111. Also, the control unit 115 may generate andtransmit a zoom-in signal or a zoom-out signal with respect to contentsdisplayed on the image display device 120 or a signal for changing alocation of the image display device 120, based on the distance betweenthe user's face and the camera 111. Also, the control unit 115 generatesand transmits a signal for rotating contents displayed on the imagedisplay device 120 or a signal for changing a location of the imagedisplay device 120, based on an angle between a line connecting theuser's two eyes and a reference line.

A detailed explanation of each element will be made with reference toFIGS. 2 through 13 below.

FIG. 2 is a flowchart illustrating a method of recognizing a location ofa user, according to an embodiment of the present invention.

Referring to FIG. 2, in step 210, the detecting unit 112 detects the twoeyes and mouth of a face of a user who is a subject to be processed byusing the camera 111. The detecting unit 112 may detect the user byperforming motion detection on an image captured by the camera 111 andskin detection in an area where a motion is detected. After detectingthe user, the detecting unit 112 detects a head, and detects the twoeyes and mouth of the head.

In step 220, the calculating unit 113 calculates a ratio of a distancebetween the two eyes to a distance between a middle point of the twoeyes and the mouth. First, the calculating unit 113 calculates adistance between the two eyes and a distance of a middle point of thetwo eyes, that is, the glabella, and the mouth. Next, the calculatingunit 113 calculates a ratio of the distance between the two eyes to thedistance between the glabella and the mouth. The ratio is obtained bydividing the distance between the two eyes by the distance between theglabella and the mouth. A distance between the glabella and the mouthwhen the user lowers his/her head may be less than a distance betweenthe glabella and the mouth when the user directly faces the camera 111,and a distance between the two eyes when the user turns his/her head maybe less than a distance between the two eyes when the user directlyfaces the camera 111.

FIG. 3 is a view for explaining a ratio of a distance between two eyesto a distance between the glabella and mouth, according to an embodimentof the present invention. Referring to FIG. 3, an x-axis represents avertical direction related to a distance between the glabella and mouth,a y-axis represents a horizontal direction related to a distance betweeneyes on the front of a face, and a z-axis represents a direction inwhich the camera 111 is looked at. In FIG. 3, a ratio refers to a ratioof a distance 301 between two eyes to a distance 302 between theglabella and mouth.

In step 230, the calculating unit 113 calculates a rotation angle of theface based on the ratio. The calculating unit 113 compares the ratiowith a reference ratio to obtain a comparison result, and calculates arotation angle of the face of the user according to the comparisonresult. In the present embodiment, the reference ratio refers to a ratioof a distance between two real eyes of the average person to a distancebetween a middle point of the two eyes and mouth. In general, a distancebetween the two eyes of a person is 6 to 7 cm, and a ratio of thedistance between the two eyes to a distance between the glabella andmouth is constantly 0.88. Although there are differences in the ratio,it is assumed in the present embodiment that the ratio is fixed at 0.88.While the ratio may vary according to the user or manufacturer, theratio is a fixed value when compared with other ratios. Further, thecalculating unit 113 calculates a ratio of a distance between the twoeyes to a distance between the glabella and mouth, and calculates arotation angle by referring to a first lookup table about a rotationangle according to a ratio, which is stored in the storage unit 114.

FIGS. 4 and 5 illustrates rotation angles of a face according toembodiments of the present invention. FIG. 6 is a view illustrating afirst lookup table showing a ratio and a rotation angle according to theratio, according to an embodiment of the present invention.

FIG. 4 illustrates a case where a user turns his/her head from a camera401. A distance between the two eyes measured by the camera 401 when theuser turns his/her head from the camera 401 is less than a distancebetween the two eyes measured by the camera 401 when the user directlyfaces the camera 401, and a distance between the glabella and mouth whenthe user turns his/her head from the camera 401 is not different from adistance between the glabella and mouth when the user directly faces thecamera 401. Accordingly, a calculated ratio is lower than a referenceratio (for example, 0.88). Referring to FIG. 6, ratios are 0.42, 0.60,and 076, and rotation angles α are 45°, 30°, and 10°, respectively.Whether the user turns his/her head to the right or to the left, arotation angle is a positive value.

FIG. 5 illustrates a case where a user lowers his/her head from a camera501. A distance between the two eyes measured by the camera 501 when theuser lowers his/her head is not different from a distance between thetwo eyes measured by the camera 501 when the user directly faces thecamera 501. A distance between the glabella and mouth when the userlowers his/her head is less than a distance between the glabella andmouth when the user directly faces the camera 501. Accordingly, acalculated ratio is greater than a reference ratio (for example, 0.88).Referring to FIG. 6, ratios are 0.90, 1.00, and 1.20, and rotationangles β are −10°, −30°, and −45°, respectively. Whether the user lowersor raises his/her head, a rotation angle is a negative value. Examplesof positive and negative values of a rotation angle are shown in FIGS. 4and 5, but may be arbitrarily set by a manufacturer.

In step 240, the calculating unit 113 detects a distance between theface and the camera 111 based on the rotation angle. The calculatingunit 113 calculates a distance between two virtual eyes when it isassumed that the user directly faces the camera 111 by using a distancebetween two calculated real eyes, a calculated rotation angle, and asecond lookup table stored in the storage unit 114. The second lookuptable shows information obtained by converting a distance between thetwo eyes into a distance between two virtual eyes when the user directlyfaces the camera based on the calculated rotation angle. Next, thecalculating unit 113 detects a distance between the face and the camera111 using the distance between the two virtual eyes and a third lookuptable. The third lookup table shows a real separation distance betweenthe face and the camera 111 according to the distance between the twovirtual eyes.

FIGS. 7 and 8 illustrate a second lookup table and a third lookup table,respectively, according to an embodiment of the present invention. Forexample, when it is assumed that a ratio measured when a user turnshis/her head as shown in FIG. 4 is 0.42, a rotation angle α iscalculated by using the first lookup table to be 45°, and when adistance between two measured real eyes is 4.5 cm, a distance betweentwo virtual eyes is calculated by using the second lookup table to be 6cm, and a distance between a real person and the camera 111 is detectedby using the third lookup table to be 1 m. When it is assumed that aratio measured when a user lowers his/her head is calculated to be 1.00as shown in FIG. 5, a rotation angle β is calculated by using the firstlookup table to be −30°, when a distance between two measured real eyesis 4.5 cm, a distance between two virtual eyes is calculated by usingthe second lookup table to be 4.5 cm, and a distance between a realperson and the camera 111 is detected by using the third lookup table tobe 2 m. Although examples of the first through third lookup tables areshown in FIGS. 6 through 8, values of the lookup tables may bedetermined experimentally.

FIG. 9 is a flowchart illustrating a method of recognizing a location ofa user, according to an embodiment of the present invention.

Referring to FIG. 9, steps 910 through 940 are similar to steps 210through 240 of FIG. 2, and thus, an explanation thereof will not begiven.

In step 950, the control unit 115 generates a zoom-in signal or azoom-out signal with respect to contents displayed on the image displaydevice 120 based on the distance between the user's face and the camera111. The control unit 115 generates the zoom-in signal or the zoom-outsignal with respect to the contents so as to provide the contents withan optimal size to a viewer of the image display device 120 andtransmits the zoom-in signal or the zoom-out signal to the image displaydevice 120, according to the distance between the user's face and thecamera 111. Next, the image display device 120 zooms in or out from thecontents according to the zoom-in signal or the zoom-out signal whichthe image display device 120 receives. Also, the control unit 115 maygenerate a zoom-in signal or a zoom-out signal with respect to theuser's face and zoom in or out from the camera 111, based on thedistance between the face and the camera 111.

In step 960, the control unit 115 generates a signal for changing alocation of the image display device 120 connected to the camera 111based on the rotation angle calculated in operation 930. If the user ofthe image display device 120 turns his/her head, the control unit 115may rotate the image display device 120 by generating a signal forlaterally rotating the image display device 120 based on the rotationangle. Next, the image display device 120 is rotated based on thesignal, which the image display device 120 receives.

FIG. 13 illustrates a rotation of an image display device 1320 accordingto a rotation angle of a face of a viewer 1350, according to anembodiment of the present invention. Referring to FIG. 13, the size ofcontents may be adaptively changed by a zoom-in signal or a zoom-outsignal according to a distance between the face of the viewer 1350 and asingle camera 1330. Also, referring to FIG. 13, a location of the imagedisplay device 1320 is changed from a location of an original imagedisplay device 1310 in such a manner that the image display device 1320is moved forward by a predetermined angle, according to a rotation angleα or β of the face of the viewer 1350.

In step 970, the calculating unit 113 calculates an angle between a lineconnecting the two eyes and a reference line. The reference line refersto a line connecting two virtual eyes when it is assumed that the userdirectly faces the camera 111. That is, if the user lowers only his/herhead, since the user still directly faces the camera 111, the rotationangle α or β is not changed. Accordingly, the ratio is constant.

FIG. 10 illustrates an angle between a line connecting the two eyes 1001and a reference line 1030, according to an embodiment of the presentinvention.

Referring to FIG. 10, the calculating unit 113 calculates an angle γbetween a line 1010 connecting the two eyes 1001 and the reference line1030.

In step 980, the control unit 115 generates a signal for changing alocation of the image display device 120 or a signal for rotatingcontents displayed on the image display device 120 based on an anglebetween a line connecting the two eyes and a reference line. Next, thecontrol unit 115 transmits the generated signal to the image displaydevice 120 or a device for controlling a location of the display device120.

FIGS. 11 and 12 illustrate rotations of contents and the image displaydevice 120 according to an angle between a line connecting the two eyesand a reference line, according to embodiments of the present invention.

Referring to FIG. 11, when a viewer 1110 of an image display device 1120lies flat to view contents 1140, an angle γ between a line connectingthe two eyes and a reference line is about 90°, and a control unitgenerates a signal for rotating the contents 1140 by about 90°. In FIG.12, a signal for rotating an image display device 1220, that is,hardware itself, by about 90° is generated. Also, in FIG. 11, the sizeof the contents 1140 is reduced by a zoom-out signal according to adistance between a face of the viewer 1110 and a single camera 1130.

According to the one or more embodiments of the present invention, evenwhen a face rotates, since a separation distance between a camera and auser may be accurately calculated using a single camera unlike in astereo method requiring two or more cameras, costs may be reduced andcalibration may not be necessary.

According to the one or more embodiments of the present invention, aninteraction effect with contents may be obtained through a cameraembedded in an image display device. For example, interactions with auser, such as a rotation of hardware, a link due to a lateral movementof contents, a change in the size of contents due to zoom-in or out ofthe contents, is achieved.

Also, a tracking system may be realized by estimating a location of auser in an image entering from a surveillance video system and operatinga Pan Tilt Zoom (PTZ) camera to directly zoom in and out.

The present invention may be embodied as computer-readable codes in acomputer-readable recording medium. The computer-readable recordingmedium may be any recording apparatus capable of storing data that isread by a computer system. Examples of the computer-readable recordingmedium include Read-Only Memories (ROMs), Random-Access Memories (RAMs),CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.The computer-readable recording medium may be a transitory or anon-transitory carrier wave that transmits data via the Internet, forexample. The computer-readable recording medium may be distributed amongcomputer systems that are interconnected through a network, and thepresent invention may be stored and implemented as computer-readablecodes in the distributed system. Functional programs, codes, and codesegments embodying the present invention may be easily derived byprogrammers in the technical field to which the present inventionpertains.

The embodiments of the present invention have been particularly shownand described. The image sensors according to the embodiments may beapplied to, for example, a light-sensitive touch panel, a digitalcamera, a camcorder, a mobile phone, and a portable electronic device.However, it will be understood by one of ordinary skill in the art thatthe embodiments should be considered in a descriptive sense only and notfor purposes of limitation, and various changes in form and details maybe made therein without departing from the spirit and scope of the oneor more embodiments of the present invention as defined by the followingclaims.

1. A method of recognizing a location of a user using a camera, themethod comprising: detecting two eyes and a mouth of the user's face;calculating a ratio of a distance between the two eyes to a distancebetween a middle point of the two eyes and the mouth; calculating arotation angle of the face according to the ratio; and detecting adistance between the face and the camera based on the rotation angle. 2.The method of claim 1, wherein calculating the rotation angle comprises:comparing the ratio with a reference ratio to obtain a comparisonresult; and calculating a rotation angle of the face of the useraccording to the comparison result, wherein the reference ratio refersto a fixed ratio of a distance between two real eyes of a person to adistance between a middle point of the two real eyes and the mouth. 3.The method of claim 2, wherein detecting the distance between the faceand the camera comprises: calculating, if it is assumed that the userdirectly faces the camera, a distance between two virtual eyes based onthe rotation angle; and detecting a distance between the face and thecamera based on the distance between the two virtual eyes.
 4. The methodof claim 3, wherein calculating the rotation angle of the face comprisescalculating the rotation angle by using a first lookup table in whichinformation about a predetermined rotation angle according the ratio isstored, calculating the distance between the two virtual eyes comprisescalculating the distance between the two virtual eyes by using a secondlookup table in which predetermined information obtained by convertingthe distance between the two eyes into a distance between two virtualeyes when the user directly faces the camera is stored, and detectingthe distance between the face and the camera comprises detecting thedistance between the face and the camera by using a third lookup tablein which information about a real separation distance between the faceand the camera according to the distance between the two virtual eyes isstored.
 5. The method of claim 3, further comprising generating azoom-in signal or a zoom-out signal with respect to the face based onthe distance between the face and the camera.
 6. The method of claim 3,further comprising a zoom-in signal or a zoom-out signal with respect tocontents displayed on an image display device connected to the camerabased on the distance between the face and the camera.
 7. The method ofclaim 2, further comprising a signal for changing a location of an imagedisplay device connected to the camera based on the rotation angle. 8.The method of claim 1, further comprising calculating an angle between aline connecting the two eyes and a reference line, wherein, when it isassumed that the user directly faces the camera, the reference linerefers to a line connecting two virtual eyes.
 9. The method of claim 8,further comprising generating a signal for rotating contents displayedon an image display device connected to the camera based on the anglebetween the line connecting the two eyes and the reference line.
 10. Themethod of claim 8, further comprising a signal for changing a locationof an image display device connected to the camera based on the anglebetween the line connecting the two eyes and the reference line.
 11. Anapparatus for recognizing a location of a user using a camera, theapparatus comprising: a detecting unit that detects two eyes and a mouthof the user's face; and a calculating unit that calculates a ratio of adistance between the two eyes to a distance between a middle point ofthe two eyes and the mouth, calculates a rotation angle of the faceaccording to the ratio, and calculates a distance between the face andthe camera based on the rotation angle.
 12. The apparatus of claim 11,wherein the calculating unit compares the ratio with a reference ratioto obtain a comparison result, and calculates the rotation angle of theface of the user according to the comparison result, and wherein thereference ratio refers to a fixed ratio of a distance between two realeyes of a person to a distance between a middle point of the real twoeyes and the mouth.
 13. The apparatus of claim 12, wherein thecalculating unit calculates, when it is assumed that the user directlyfaces the camera, a distance between two virtual eyes based on therotation angle, and detects a distance between the face and the camerabased on the distance between the two virtual eyes.
 14. The apparatus ofclaim 13, further comprising a storage unit that stores a first lookuptable in which information about a predetermined rotation angleaccording to the ratio is stored, a second lookup table in whichinformation obtained by converting the distance between the two eyesbased on the rotation angle into a distance between two virtual eyeswhen the user directly faces the camera is stored, and a third lookuptable in which information about a real separation distance between theface and the camera according to the distance between the two virtualeyes is stored, wherein the calculating unit calculates the rotationangle by using the first lookup table, calculates the distance betweenthe two virtual eyes by using the second lookup table, and detects thedistance between the face and the camera by using the third lookuptable.
 15. The apparatus of claim 13, further comprising a control unitthat generates a zoom-in signal or a zoom-out signal with respect to theface based on the distance between the face and the camera.
 16. Theapparatus of claim 13, further comprising a control unit that generatesa zoom-in signal or a zoom-out signal with respect to contents displayedon an image display device connected to the camera based on the distancebetween the face and the camera.
 17. The apparatus of claim 12, furthercomprising a control unit that generates a signal for changing alocation of an image display device connected to the camera based on therotation angle.
 18. The apparatus of claim 11, wherein the calculatingunit calculates an angle between a line connecting the two eyes and areference line, and wherein, when it is assumed that the user directlyfaces the camera, the reference line refers to a line connecting twovirtual eyes.
 19. The apparatus of claim 18, further comprising acontrol unit that generates a signal for rotating contents displayed onan image display device connected to the camera based on the anglebetween the line connecting the two eyes and the reference line.
 20. Theapparatus of claim 18, further comprising a control unit that generatesa signal for changing a location of an image display device connected tothe camera based on the angle between the line connecting the two eyesand the reference line.